We investigate the dynamics of sensory integration for perceiving musical performance, a complex natural behavior. Thirty musically trained participants saw, heard, or both saw and heard, performances by two clarinetists. All participants used a sliding potentiometer to make continuous judgments of tension (a measure correlated with emotional response) and continuous judgments of phrasing (a measure correlated with perceived musical structure) as performances were presented. The data analysis sought to reveal relations between the sensory modalities (vision and audition) and to quantify (...) the effect of seeing the performances on participants' overall subjective experience of the music. In addition to traditional statistics, functional data analysis techniques were employed to analyze time-varying aspects of the data. The auditory and visual channels were found to convey similar experiences of phrasing but different experiences of tension through much of the performances. We found that visual information served both to augment and to reduce the experience of tension at different points in the musical piece (as revealed by functional linear modeling and functional significance testing). In addition, the musicians' movements served to extend the sense of phrasing, to cue the beginning of new phrases, to indicate musical interpretation, and to anticipate changes in emotional content. Evidence for an interaction effect suggests that there may exist an emergent quality when musical performances are both seen and heard. The investigation augments knowledge of human communicative processes spanning language and music, and involving multiple modalities of emotion and information transfer. (shrink)

To understand how individuals adapt to and anticipate each other in joint tasks, we employ a bidirectional delay–coupled dynamical system that allows for mutual adaptation and anticipation. In delay–coupled systems, anticipation is achieved when one system compares its own time‐delayed behavior, which implicitly includes past information about the other system’s behavior, with the other system’s instantaneous behavior. Applied to joint music performance, the model allows each system to adapt its behavior to the dynamics of the other. Model predictions of asynchrony (...) between two simultaneously produced musical voices were compared with duet pianists’ behavior; each partner performed one voice while auditory feedback perturbations occurred at unpredictable times during live performance. As the model predicted, when auditory feedback from one musical voice was removed, the asynchrony changed: The pianist’s voice that was removed anticipated (preceded) the actions of their partner. When the auditory feedback returned and both musicians could hear each other, they rapidly returned to baseline levels of asynchrony. To understand how the pianists anticipated each other, their performances were fitted by the model to examine change in model parameters (coupling strength, time‐delay). When auditory feedback for one or both voices was removed, the fits showed the expected decrease in coupling strength and time‐delay between the systems. When feedback about the voice(s) returned, the coupling strength and time‐delay returned to baseline. These findings support the idea that when people perform actions together, they do so as a coupled bidirectional anticipatory system. (shrink)

High-level improvising musicians master idiosyncratic gesture vocabularies that allow them to express themselves in unique ways. The full use of such vocabularies is nevertheless challenged when improvisers incorporate electronics in their performances. To control electronic sounds and effects, they typically use commercial interfaces whose physicality is likely to limit their freedom of movement. Based on Jim Black's descriptions of his ideal digital musical instrument, embodied improvisation gestures, and stage performance constraints, we develop the concept of a modular wearable MIDI interface (...) to closely meet the needs of professional improvisers, rather than proposing a new generic instrument that would require substantial practice to adapt improvisational techniques already acquired. Our research draws upon different bodies of knowledge, from theoretical principles on collaboration and embodiment to wearable interface design, in order to create a digital vest called Track It, Zip It that features two innovative on-body sensors. Allowing for sound control, these sensors are seamlessly integrated with Black's improvisational gesture vocabulary. We then detail the design process of three TIZI prototypes structured by the outcomes of a performance test with Black, a public performance by a novice improviser during the 2017 International Guthman Musical Instrument Competition, and measurements of sensor responses. After commenting on the strengths and weaknesses of the final TIZI prototype, we discuss how our interdisciplinary and collective process involving a world-class improviser at the very center of the design process can provide recommendations to designers who wish to create interfaces better adapted to high-level performers. Finally, we present our goals for the future creation of a wireless version of the vest for a female body based on Diana Policarpo's artistic vision. (shrink)